Repository navigation
A cluster-info outage never stops a managed Worker - #451
Conversation
Digest re-review: any failure of GET /api/cluster/info killed the PHP Worker. Every workflow-task completion (and activity completion with payloads) reads it for the payload policy, at most once a minute; a plain 500 there ended Worker::run(), and startup discovery retried only the narrow untyped-outage set without reporting a lasting outage. Cluster info is read-only, so TransientFailure::isReadOutage() treats a refused, reset or timed-out connection and any 5xx or 429 (unless retryable: false) as an outage; 4xx refusals stay fatal. - RuntimePayloadUploads keeps the last discovered policy through an outage and tries discovery again 10 s later instead of before every request. With no policy yet, a Worker request fails as a typed not-admitted payload_discovery_unavailable (nothing was sent), which the Worker repeats with the shared jittered schedule and reports through worker.outage_persisting at 60 s, then every 5 minutes. The exact activity-completion envelope keeps its existing handling. - Startup discovery (cooperative cancellation, query authority) retries read outages the same way and reports a lasting one; it waits like registration. Tests over the real Psr18Transport (500, refused connection with cURL errno 7, bodiless 502): a workflow-task completion waits out 30 failed reads, is sent once, nothing is reported as a task failure, and the Worker keeps running; a cooperative Worker started during an outage registers once discovery answers. All fail on main. A unit test covers the stale policy fallback. Co-Authored-By: Claude Opus 5.5 <[email protected]>
Review blocker: the Server's inline limit was lowered to 100 KB while the Worker held a 2 MB payload policy. A 500 KB activity result was sent inline, the Server answered 413 payload_too_large, the refusal was uncaught and the Worker exited (the activity then failed as having lost its lease). The stale-policy fallback from the previous commit widens that window; a fresh 60 s cache already allowed it. Client: on a 413 payload_too_large for an activity or workflow-task completion (refused before the body was read, so nothing was recorded), drop the cached policies, read cluster info again, encode the same outcome under the new policy (externalising what it requires) and send it once more. If re-encoding changes nothing, the refusal stands. A discovery outage at that point fails as not sent, and the Worker repeats the completion under the outage rules without recomputing the result. Worker: an activity result that is still refused for its size is the attempt's failure, not the Worker's: it is reported as a non-retryable ActivityResultTooLarge and the Worker keeps running. The cluster-info outage tracker is now separate, so polls that keep succeeding no longer reset it and a discovery outage spanning several claims still escalates. Tests (real Psr18Transport, server-side limit change mid-run): the result is externalised and completed once with one rediscovery; a rediscovery outage is retried without recomputing; an outage outlasting the claim leaves the attempt to the Server and the re-delivered attempt completes; a result too large even under the new limit fails the attempt non-retryably. Existing genuine-413 tests now expect the one policy re-read before the refusal stands. Co-Authored-By: Claude Opus 5.5 <[email protected]>
Rust (#306) reports a still-too-large result with failure.type and failure.class "ActivityResultTooLarge"; PHP sent its fully qualified class name. Send the same plain string in both fields, so a workflow sees one reason code whichever SDK ran the activity (Python #360 matches). Co-Authored-By: Claude Opus 5.5 <[email protected]>
|
rmcdaniel
left a comment
There was a problem hiding this comment.
Reviewed the coordinated worker payload-limit fix against the PHP #451, Python #360 and Rust #306 changes. PHP, Python and Rust recover only from the Server's typed size refusal. The worker can rediscover the policy and re-encode the result without re-running the activity. A result that still cannot fit is reported with type and class ActivityResultTooLarge and non_retryable: true, and worker dispatch remains available. A discovery outage keeps the attempt eligible for recovery instead of recording a permanent size failure.
The target-branch and language CI jobs passed for all three reviewed heads. Python also passed the local focused transport suite, and Rust passed the 17 worker-outage tests. The release qualification and published-package checks follow the merges.



From the final digest re-review. Before this change, any failure of
GET /api/cluster/infostopped the PHP Worker.Why
Every workflow-task completion reads cluster info for the payload policy, at most once a minute (
RuntimePayloadUploads::policy()). So does an activity completion that carries payloads. A plain 500 on that read endedWorker::run()before the completion was ever sent. Startup discovery, used for cooperative cancellation and query authority, retried only the narrow untyped-outage set and never reported a lasting outage.Change
Cluster info is read-only and always safe to repeat.
TransientFailure::isReadOutage()treats these as an outage: a refused, reset or timed-out connection, and any 5xx or 429 unless it saysretryable: false. 401, 403 and other 4xx answers stay fatal.RuntimePayloadUploadskeeps the last discovered policy through an outage and tries discovery again 10 s later, not before every request. The Server validates every request anyway. With no policy yet, a Worker request fails as a typedpayload_discovery_unavailablewithrequest_admitted: false, because nothing was sent. The Worker repeats it with the shared jittered backoff (0.25 s doubling to a 4 s cap) and reports a lasting outage asworker.outage_persisting: once at 60 s, then every 5 minutes. If the completion retries run out,run()keeps going and the Server re-delivers the task when its lease ends. The exact activity-completion discovery envelope keeps its existing handling.Tests
All of these run over the real
Psr18Transport. The failure cases are a 500, a refused connection reported as cURL errno 7, and a bodiless 502. Every one of them fails on main.worker.outage_persistingis logged, and there is noworker.fatal_failure.retryable: false, 401, 403 and 404 are not.testDiscoveryRetryClassificationRequiresAnExactPreSubmissionBackendEnvelopeis updated. A 5xx from discovery is now the generic not-admitted outage, still not the exact activity envelope, and a 401 surfaces as itself.Results: the full suite passes (9229 tests, 56 skipped), and phpstan on
srcis clean. Docs (docs/portal/build/workers.md) and the CHANGELOG are updated.The matching PRs are sdk-python#359 (startup) and sdk-rust#306.
Review fix: a stale policy and a lowered limit (commit cf87771)
In review, the Server's inline limit was lowered to 100 KB while the Worker held a 2 MB policy. A 500 KB result went out inline and got
413 payload_too_large. That error was not caught, so the Worker exited.Client: when a completion is refused with a 413 before its body is read, nothing was recorded. The client drops the cached policies, reads cluster info again, encodes the same outcome under the new policy (externalising what it requires), and sends it once more. If re-encoding changes nothing, the refusal stands. If discovery is down at that point, the completion counts as not sent, and the Worker retries it under the outage rules without recomputing the result.Worker: an activity result that is still too large is reported as a non-retryableActivityResultTooLargefailure, and the Worker keeps running. The cluster-info outage is now tracked separately, so polls that succeed no longer reset it.Tests:
WorkerStalePayloadPolicyTestchanges the server-side limit mid-run. It covers four cases:Existing genuine-413 tests now expect the single policy re-read.
🤖 Generated with Claude Code